Lightweight Neural Networks

Most of the weights in a Lightweight Neural Network have a value of zero, while the remaining ones are either +1 or −1. These universal approximators require approximately 1.1 bits/weight of storage, posses a quick forward pass and achieve classification accuracies similar to conventional continuous-weight networks. Their training regimen focuses on error reduction initially, but later emphasizes discretization of weights. They ignore insignificant inputs, remove unnecessary weights, and drop unneeded hidden neurons. We have successfully tested them on the MNIST, credit card fraud, and credit card defaults data sets using networks having 2 to 16 hidden layers and up to 4.4 million weights. 1 Lightweight Neural Networks Lightweight Neural Networks (LWN) are a subset of the conventional Continuous-Weight Networks (CWN). We call them lightweight because the trained LWNs have weights that require approximately 1.1 bits/weight of storage and their forward-passes does not require floating-point multiplications. The key characteristic of LWNs is the sparsity of their weight matrices. Moreover, the non-zero weights of these matrices are limited to only two values: ±1 (see Figure 1). These networks were first introduced in 1996 [1, 2] as Multiplier-Free Networks and used training heuristics that were proposed in 1994 [3]. Due to the recent interest in similar networks [4, 5, 6, 7], we present new results highlighting the sparsity of these networks, their natural inclination towards forming tight receptive fields, and their universal approximation capability. 2 LWN and the Biological Neural Network Here we would like to highlight those aspects of the LWN neurons that make them more similar in structure and function to the biological neurons as compared with the CWN neurons. Consider an axon of a source biological neuron connecting to the dendrite of the target neuron at a synapse. One of two types of neurotransmitter chemicals (either for excitatory or inhibitory receptors) is released from the axon’s side of the synapse whenever the source neuron is activated [8]. These chemicals then bind with receptors on the dendrite-side of the synapse, resulting in an increase (in case of an excitatory receptor) or decrease (inhibitory receptor) in the electrical potential on the membrane of the target neuron. The electrical potential of that membrane is the sum of contributions due to the firings of all neurons that are connected through synapses to the target neuron. When the membrane’s electrical potential reaches a threshold value, the target neuron fires. The highlight of the above narrative is the absence of multiplication operations and the presence of only two synaptic values, excitatory (similar to the +1 weight of LWN connections) and inhibitory(−1 weight)1. The connections-to-neurons ratio decreases with increase in the number of neurons in biological systems [9, 10]. LWNs exhibit the same characteristic (Figure 2), but CWNs can not. The size of the 1LWN neurons are different from the biological neuron in having bipolar outputs as well as bipolar inputs. In case of the negative-valued inputs, the roles of excitatory and inhibitory receptors are reversed. 1 ar X iv :1 71 2. 05 69 5v 1 [ cs .L G ] 1 5 D ec 2 01 7